What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment∗

Due to the inherent sensitivity of many survey questions, a number of researchers have adopted an indirect questioning technique known as the list experiment (or the item count technique) in order to minimize bias due to dishonest or evasive responses. However, standard practice with the list experiment requires a large sample size, is not readily adaptable to regression or multivariate modeling, and provides only limited diagnostics. This paper addresses all three of these issues. First, the paper presents design principles for the standard list experiment (and the double list experiment) to minimize bias and reduce variance as well as providing sample size formulas for the planning of studies. Additionally, this paper investigates the properties of a number of estimators and introduces an easy-to-use piecewise estimator that reduces necessary sample sizes in many cases. Second, this paper proves that standard-procedure list experiment data can be used to estimate the probability that an individual holds the socially undesirable opinion/behavior. This allows multivariate modeling. Third, this paper demonstrates that some violations of the behavioral assumptions implicit in the technique can be diagnosed with the list experiment data. The techniques in this paper are illustrated with examples from American politics. ∗An earlier version of this paper was presented at the 2010 Midwest Political Science Association conference. The author would especially like to thank Adam Berinsky for providing the list experiment data that was used in this paper as well as his collaboration in composing the items in the list experiment. These data were collected in conjunction with the 2010 Political Experiments Research Lab (PERL) Omnibus Study (Adam Berinsky PI). The author would also like to thank Matt Blackwell, Richard Nielsen, Justin Grimmer, Chase Harrison, Sunshine Hillygus, Gary King, and Kevin Quinn for their helpful comments and suggestions. The usual caveat applies. †Department of Government and The Institute for Quantitative Social Sciences Harvard University, 1737 Cambridge Street, Cambridge, MA 02138. [email protected]